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Founded in 1900, the Association of American Universities comprises 62 distinguished institutions in 
the United States and Canada that continually advance society through education, research, and 
discovery. Our U.S. member universities earn the majority of competitively awarded federal funding 
for academic research, are improving human life and well-being through research, and are educating 
tomorrow’s visionary leaders and global citizens. AAU members collectively help shape policy for 
higher education, science, and innovation; promote best practices in undergraduate and graduate 
education; and strengthen the contributions of research universities to society. 
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The mission of the Pullias Center for Higher Education is to bring a multidisciplinary perspective to 
complex social, political, and economic issues in higher education. Since 1996, the center has 
engaged in action-oriented research projects regarding successful college outreach programs, 
financial aid and access for low- to moderate-income students of color, use of technology to 
supplement college counseling services, effective postsecondary governance, emerging 
organizational forms such as for-profit institutions, and the retention of doctoral students of color. 
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Introduction 


Increasingly, institutions of higher education are facing numerous complex challenges. These 
include: 


= expanding access; 

=" supporting success among first-generation, low-income, historically underrepresented 
minority groups, and other underserved populations; 

= implementing evidence-based teaching practices, assessment, and new technologies; 

= improving the quality of curriculum through service-learning, international and global 
perspectives and interdisciplinarity; 

= addressing on-going challenges with remedial education, student transition, and inclusion; 

= cost-cutting for improved affordability; 

= altering hiring and contracts for faculty and staff to create conditions that allow them to 
better support students; and 

= jntegrating new practices or topics like sustainability that support prosocial goals and 
provide an example for students of real-world connections to the curriculum. 


Many external groups, such as foundations or national associations, have chosen a particular issue or 
area in which they want to support higher education institutions as they attempt to make progress 
addressing these complex challenges. This guide is for these key external stakeholder groups 
(policymakers, national associations, reform groups, accreditors, foundations, government agencies, 
business and industry) that support change and reform of the higher education enterprise. It presents 
a toolkit that provides advice for external stakeholders on how to be effective partners in working with 
colleges and universities to support changes that go to scale. National organizations often work directly 
with institutions so we will be mentioning institutions from time to time as part of the strategy. 


Scale is typically defined as a reform or change 
affecting more than just small group of students 
and often involving multiple departments, units, 
and institutions. In the guide we refer to scale up, 
scaling or scale, which all refer to this same process. 


Scale is typically defined as a reform or change affecting more than just a small group of students 
and often involving multiple departments, units, and institutions. Goals around scale may differ 
substantially by stakeholder organization. While no longer the norm, some groups work primarily 
with individual institutions and consider a practice to be scaled when it becomes widespread at a 
single campus. Stakeholders working on these types of efforts are already well-served; many guides, 
including Association of American Colleges & Universities’ /ncreasing Student Success in STEM and 
Adrianna Kezar’s How Colleges Change, exist to support change on individual campuses. 


Alternatively, some foundations or agencies work with a set of campuses, sometimes joined through 
consortia, other times through multi-campus projects. This guide will be helpful for such multi- 
campus efforts, as will Scaling and Sustaining Change and Innovation, a report by Kezar sponsored by 
the Teagle Foundation that focuses on lessons learned about implementing and sustaining changes 
among 10 consortium-funded projects that involved close to a hundred campuses. 
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However, this guide will be especially valuable for external stakeholders who are aiming for large- 
scale change of practice or policy across multiple institutions or the entire higher education sector. 
Foundations, agencies and non-profits are hoping to foster change at an even broader scale that 
will not just affect a set of campuses, but rather an entire sector, a discipline, or all of higher 
education. Such ambitious efforts include Vision and Change in Undergraduate Biology Education 
and Partnership for Undergraduate Life Sciences Education (PULSE), which works to change 
curriculum and teaching across all of biology. Similar efforts are also occurring in physics with the 
Strategic Programs for Innovation in Undergraduate Physics (SPIN-UP) project and through 
accreditation in engineering. Additionally, accreditors and the National Institute for Learning 
Outcomes Assessment are currently working to achieve scale in the use of student learning 
outcomes and assessment. 


While this guide is focused on improving teaching and learning, the lessons can be 
used by organizations to scale other forms of change. 


To demonstrate an example of such large-scale change efforts in action, we share lessons learned 
from the Association of American Universities (AAU) Undergraduate STEM Education Initiative (see 
p. v). This project has the goal of scaling evidence-based teaching practices across AAU institutions 
(top research universities) and teaming up with others working to improve undergraduate STEM 
education to make excellent teaching a norm in research universities. 


This guide evolved from the study of the AAU Initiative, and can be used in tandem with the larger, 
final project report, Scaling Improvements in STEM Learning Environments: The Strategic Role of a 
National Organization, which provides more detail on the concepts that follow. This guidebook 
provides specific tools and advice for stakeholder groups to create a strategic approach to scaling 
change. In short, this guide will help organizations understand critical strategies informed by 
theories of change: 


assess organizational strengths and weaknesses 
ensure distributed leadership 

evaluate framing and language for change 
utilize multiple theory-based strategies 

create and assess a systems approach 

leverage influence strategies 

build and support networks 

create feedback loops 
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While we suggest that action in each of these areas is important for scaling change, an 
organization may choose to focus on and improve just a few strategies. We begin with an overview 
of theories of change and then outline eight strategies organizations can use to facilitate change. 
This is not a linear process and any step can be engaged in any order. However, we recommend that 
organizations first identify and develop the appropriate leadership or language (steps 2 and 3), before 
tackling influence strategies and networking (steps 6 and 7). Once the right framing and leadership 
are in place, networks will likely be easier to develop. 
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We also provide concrete prompts in each section, yet one need not follow each step in this guide 
to be successful. The prompts are merely questions to help think through an influence strategy or 
method of deploying multiple theories of change to make these processes more concrete for those 
needing such guidance; one can also be successful by considering these issues more conceptually. 
In the end, reflection on these processes will enhance one’s approach to scale. 


Strategies for Scaling Change 


Setting the context for Scaling practices Refining scaling practices 
scale 
= Asset assessment = Multi-theory = Feedback loops 
= Distributed = Systems 
leadership = Influence 
= Language and framing = Networks 


About the AAU Undergraduate STEM Education Initiative 


In 2011, the Association of American Universities (AAU) launched a five-year initiative in partnership 
with member institutions to improve undergraduate teaching and learning in science, technology, 
engineering and mathematics (STEM) fields that moves beyond individualistic approaches (e.g. 
single classrooms or departments) to change. 


The overall objective of the AAU Undergraduate 
STEM Education Initiative is to influence the culture 
of STEM departments at AAU universities so that 
faculty are encouraged and supported to use 
teaching practices proven by research to be more 
effective in engaging students in STEM education 
and in helping students learn. 


The overall objective of the AAU Undergraduate STEM Education Initiative is to influence the culture 
of STEM departments at AAU universities so that faculty are encouraged and supported to use 
teaching practices proven by research to be more effective in engaging students in STEM education 
and in helping students learn. The goals of the Initiative are to: 


1. Develop an effective analytical framework for assessing and improving the quality of 
STEM teaching and learning; 

2. Support project sites at a subset of AAU universities to implement the Framework, and 
develop a broader network of AAU universities committed to implementing STEM 
teaching and learning reforms; 
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3. Explore mechanisms that institutions and departments can use to train, recognize, and 
reward faculty members who want to improve the quality of their STEM teaching; 

4. Work with federal research agencies to develop mechanisms for recognizing, rewarding, 
and promoting efforts to improve undergraduate learning; and 

5. Develop effective means for sharing information about promising and effective 
undergraduate STEM education programs, approaches, methods, and pedagogies. 


In collaboration with its member universities, AAU developed a Framework for Systemic Change in 
Undergraduate STEM Teaching and Learning (“the Framework,” see Figure 1) to guide institutions in 
their commitment to facilitate change in undergraduate STEM education. AAU selected eight 
member campuses to serve as demonstration project sites. Over three years, each of the eight 
project sites implemented a major undergraduate STEM education project that incorporated key 
elements of the Framework, including: pedagogy—implementing and assessing the efficacy of 
research-based pedagogies; scaffolding—supporting faculty learning and development (e.g. 
providing a center for teaching and learning, enhanced classrooms); and cultural change—working 
to change policies and practices that are not supportive of undergraduate teaching (e.g. tenure and 
promotion policies). Ultimately, these project sites served as laboratories to implement the 
Framework and they are the first phase in an effort to encourage broad-based reform of STEM 
undergraduate teaching practices at AAU research universities and beyond. 


Figure 1 
The Framework 


CULTURAL CHANGE 


SCAFFOLDING 
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AAU is also actively bringing together more campuses from among its members to form an AAU 
STEM Network. The AAU staff envision a collaborative network that will help to support and link 
AAU institutions grappling with similar challenges and barriers in reforming and improving STEM 
teaching and learning for undergraduate students. Complementing these efforts is AAU’s work on 
metrics. AAU developed a set of baseline measures (e.g. how many courses involve technology or 
active learning) of the Framework, such as use of evidence-based teaching practices and 
professional development offered that project sites (and other institutions) may use to better 
understand the current status of teaching and learning and to begin documenting progress. These 
measures align with the Framework, as they ask about changes in introductory courses, 
development of more active learning classrooms, faculty learning communities, or changes in 
tenure and promotion policies. 


The research project that generated this guide followed the AAU Initiative in real time; the 
researchers observed Initiative meetings and events, then interviewed faculty, administrators and 
collaborators with the Initiative about their perceptions of what they felt was working to scale 
change as well as what might not be working as effectively. The project involved three years of 
observation, review of thousands of pages of documents, and interviews with over 110 people. The 
final report from the project can be found at pullias.usc.edu/scalingstemreform. 


Scaling STEM Reform Project Data 


3 years of observations 110+ interviews Thousands of pages 
at AAU Initiative with faculty administrators, of documents reviewed 
meetings and events and collaborators with 
Initiative 


pullias.usc.edu/scalingstemreform 
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Theories of Change 


Before considering a strategy for scaling change, it is important to become familiar with various 
theories or approaches to change in order to ensure alignment between theory and action. Theories 
of change are explanations of how change occurs or progresses in an organization, and thus helpful in 
illustrating the potential ways change may unfold. And in fact, funding organizations are increasingly 
asking for proposals that articulate a theory of action around how leaders will implement their 
project and what evidence exists that such an approach to change will work. An organization might 
use one or several of these strategies. 


We will refer to these theories throughout this guide, so we introduce them here first. Specifically, 
we describe six sets of theories that Kezar has found undergird change processes in higher 
education, particularly those focused on scale: institutional theories, network theories, theories of 
organizational learning, cultural theories, political theories, and systems theory/ institutionalization. 
Each theory of change has different assumptions about the key levers that drive change and the 
necessary actions that should accompany them. For more details see Kezar's book, How Colleges 
Change: Understanding, Leading, and Enacting Change, 2nd Edition (2018). 


A. Institutional Theory/Influence 

Institutional theory (IT) emphasizes change as a result of external forces and the reshaping of 
cultural norms and logics. IT describes the impact of broader forces and organizations outside of 
college campuses on change, including accreditation, economic changes, disciplinary societies, 
corporatization of campuses, state policymakers, neoliberalism, and national agencies such as the 
National Science Foundation (Powell & DiMaggio, 1991). These external forces and organizations are 
known as fields. The societal field encompasses broader societal forces like economic changes that 
are more remote yet still shape campus actions. The organizational field is composed of organizations 
that are more directly tied to higher education (such as accreditors, foundations, or national 
associations), which can also help deliver and temper societal forces. Examples of how forces in the 
organizational field shape change include: national associations pushing for a more similar general 
education curriculum across institutions; accreditors pushing for assessment of student learning 
outcomes; or presidents altering their institutions’ missions to be more research-oriented in pursuit 
of prestige to be more like elite institutions (Boyce, 2003). 


IT leverages legitimacy and influence as ways to motivate change. This lever is often seen in 
isomorphism, where institutions mimic each other and grow more similar; this phenomenon is 
particularly apparent when less prestigious institutions mimic more prestigious ones (Scott, 2008; 
Taylor & Morphew, 2010). But influence can take many other forms, as well, such as normative 
approaches where certain values or approaches are legitimized, such as the recent widespread 
increase in awareness of diversity and inclusion issues on college campuses (Scott, 2008). In these 
normative approaches, shaping norms, language, discourse, and logics that underlie organizations 
can bea lever of influence for external organizations interested in promoting change. Change, when 
it occurs, is a result of a new schema or set of norms, embedded in language, transferred through 
discourse and entrenched in institutional structures over time through a system of institutional 
logics—a driving rationale for how things should operate (Scott, 2008). For example, the AAU 
Initiative attempted to build anew norm around teaching excellence at research universities through 
discourse and framing—by using language suggesting that AAU institutions should be “as excellent 
in teaching as in research.” Legitimacy is also critical in that only certain players in the field are seen 
as being able to drive new norms, standards, and logics (Scott, 2008). In higher education, these are 
often the most prestigious colleges and universities or the most prominent national organizations. 
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Historically, various studies have demonstrated how elite research universities such 
as Harvard served as the gold standard for other institutions, which began to mimic 
the research activities of these elite universities. The AAU Initiative aimed to utilize 
the isomorphic power of research universities to shift and change the overarching 
norms about research within higher education to balance them more equally between 
teaching and research, and to institutionalize the notion that faculty should be as 
excellent in teaching as they are in research. 


B. Network Theories 

Networks (sometimes called social networks) are groups of people with some sort of specified 
relationship or tie; networks can be composed of people who work in the same organization or 
industry, or who know each other socially or through shared activities (Borgatti & Halgin, 2011). 
In terms of promoting change, networks can function to build relationships, connect various 
previously disparate aspects of a system, disseminate information, facilitate learning, and eventually 
scale change. Many empirical studies demonstrate that changes in behaviors and mindsets have 
been a result of networked diffusion models in which innovators began utilizing new practices, and 
then after some success, others began to become aware of the practice and utilize it as well (Rogers, 
2003; Tenkasi & Chesmore, 2003). Networks create the informational channels for change ideas to 
flow, a community to innovate in safe spaces, and intellectual capital and knowledge about how to 
implement change (Tsai, 2002). Webs of relationships are often the chief determinants of how well 
and how quickly change efforts take hold, diffuse, and are sustained (Daly, 2010). 


Researchers have identified several key ways that networks lead to change. First, networks offer a set of 
mechanisms that enable change—through communication systems, knowledge transfer, alteration 
of schema or mindset, shaping of attitudes, increasing of problem-solving, and accountability (Ahuja, 
2000; Borgatti & Foster, 2003; Kraatz, 1998; McGrath & Krackhardt, 2003; Szulanski, 1996; Wasserman 
& Faust, 1994). Second, two specific outcomes of social network formation have been related to 
change—learning and social capital (Borgatti & Foster, 2003; Burt, 2000; Kilduff & Tsai, 2003; Tenkasi 
& Chesmore, 2003). The relationships in social networks promote learning among network members, 
and learning has been strongly linked to changes in behavior (Tenkasi & Chesmore, 2003). Networks 
also provide social capital that facilitates the change process (Burt, 2000), varying from knowledge 
about how the organization works to influence to finances. Third, change often involves risk-taking 
that can be less problematic if it is done collectively rather than individually (Valente, 1995). If one’s 
peers are engaging in the same behavior, then one is more likely to also engage in this behavior 
(Rogers, 2003; Valente, 1995). 


The AAU Initiative leveraged the power of its network of member institutions to 
promote learning and provide an environment for peer institutions to undertake 
a potentially risky change process together. The Initiative created a network of 
networks by linking the AAU network (and its subnetworks of department chairs, 
deans, provosts, presidents, and centers for teaching and learning) to other STEM 
reform networks. It also joined a coalition called CRUSE—Coalition for Reform of 
Undergraduate STEM Education, a group of national associations working to improve 
the effectiveness of undergraduate education. 
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C. Organizational Learning Theory 

Theories of organizational learning suggest that learning is a key driver for change (Senge, 1990). 
Without people challenging existing assumptions, reviewing new information, and undergoing an 
inquiry process that leads to a consideration of new ways to conduct work, change can be difficult. 
If learning is missing, actors will lack the knowledge to adopt new practices. Organizational learning 
involves information available within organizations, the way that information is shared, facilitative 
mechanisms for learning such as teams, networks (as described above) or data dashboards, sharing 
of best practices, and more generally looking at the ways that individuals within organizations 
examine or understand concepts. Organizational learning theories focus on issues such as acquiring 
information and ideas, interpreting data, turning information to knowledge, knowledge recall and 
memory, and ways to sustain learning by embedding it into the organizational structures, often 
termed knowledge management (Argyis & Schon, 1996; Garvin, 1993). Studies from multiple 
disciplines demonstrate the infrastructure that needs to be in place to support organizational 
learning (e.g. improved data systems, knowledge management systems), best practices for fostering 
learning (e.g. better formats for displaying information to different stakeholders, team practices 
that help interpret information to create knowledge), and the leadership and culture (e.g. creating 
a culture of trust, leaders supporting risk-taking and innovation) required to ensure organizational 
learning occurs (Argyis & Sch6n, 1996; Kezar, 2005). 


The AAU Initiative employed a number of actions that fit within organizational learning 
theories of scaling change. For one, the AAU leadership facilitated learning through 
its networks, allowing members to adopt practices from each other’s campuses. 
To that end, the eight demonstration sites served as laboratories that tried out new 
practices, so that the results of those efforts could inform and be adopted by other 
campuses. Moreover, four out of the Initiative’s five main objectives—including 
developing meetings for information sharing about the best practices for STEM 
reform, working with institutions and departments to support faculty, and supporting 
project sites to develop and implement the framework—related to organizational 
learning. The Initiative was also informed by the collection of data. Surveys were 
conducted to determine the extent to which evidence-based teaching practices were 
used on member campuses, understand practices to assess teaching in promotion 
and tenure, and assess learning spaces. These surveys helped fulfill a major goal of 
the Initiative, which was to develop measures that could help institutions determine 
their progress and learn how to improve implementation by use of data to inform 
their actions. 


D. Cultural Change Theories 

Culture change happens when the underlying values and assumptions of a campus change. New 
values are reflected in the practices, policies and structures of the organization (Kezar, 2013; Kuh & 
Whitt, 1988). Much of the early literature focused on culture change at the institutional level focused 
on leaders and how their priorities, language, discourse and actions shaped culture; in these studies, 
change occurred as a result of shifting leaders’ priorities, language, discourse and actions to reflect 
the desired culture (Schein, 2010). More recent literature has shown that change agents can work 
to change culture at multiple levels of the organization (Kezar, 2013). Culture change can also 
create deeper changes that are more permanent because culture change works at changing the 
unconscious assumptions that drive behaviors within organizations and systems (Kegan & Lahey, 
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2009). Strategies for change agents to create culture change range from vision setting, discussion of 
mission and values, explorations of existing values, creation of new rituals and symbols, alteration 
of language and communications to reflect new values, and leaders’ efforts to communicate new 
values (Kezar, 2103). 


AAU utilized the cultural change approach by attempting to alter the value system 
of STEM departments to realize the potential of evidence-based teaching practices, 
new curriculum, and student learning goals and outcomes. 


E. Political Theories 

Political theories examine how collective action can be leveraged for change vis-a-vis coalitions, 
development of allies, and partnerships. Strategies include agenda-building, bargaining, 
consciousness-raising, campaigns, and persuasion processes (Bolman & Deal, 2017; Kezar, 2013). The 
political approach can turn into a social movement if it moves to a widespread, more collective effort. 
It often begins as grassroots activism, but over time can develop into a full movement involving vast 
numbers of people. Political approaches center on use of power, persuasion, and communication 
to sway changes. They also focus on aligning interests of diverse groups, assuming individuals have 
differing interests. 


AAU used the political approach by tailoring communication strategies to various 
groups, such as faculty, deans, and senior leaders, while also working to build 
consensus around the importance of excellence in undergraduate STEM education. 
AAU appealed to faculty members’ interest in disciplinary strategies and evidence 
around changing teaching practices and appealed to deans and senior leaders’ 
public concerns about the affordability and quality of undergraduate education and 
declining state funding. 


F. Systems Theory and Institutionalization Theory 

Systems theory examines the interrelationship of various subsystems within an organization and the 
interconnections between organizations. In order to change teaching and promote more evidence- 
based teaching practices, professional development alone is insufficient, as classroom practices 
are also tied to incentive systems, departmental norms, facilities, campus priorities, and student 
expectations, for example. In systems theory, change is most likely to be achieved when all aspects 
of the system are adjusted. Many studies of change adopt a systems approach, exploring internal 
mechanisms like reward systems and policies as a way to influence change (Kezar, 2013). Systems 
theory suggests an interrelationship of various aspects—tenure and promotion requirements, 
professional development, institutional commitment, and classroom assessment. It also suggests 
how policies and practices need to be altered in support of a change. Institutionalization theory 
suggests that the institutional infrastructure and leadership must support changes for them to scale 
and be sustained. Institutionalization can also be seen as a facet of systems theory itself, in which the 
institution is a system and various parts need attention in order to be altered. This theory emphasizes 
incentive and reward structures in particular. But this theory suggests the importance of aligning 
policies and practices to stated change direction. 
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The systems approach is evident in the AAU Initiative's effort to operate at multiple 
levels: (1) project demonstration sites; (2) the AAU network; (3) work with federal 
agencies; (4) and the broader higher education system. The most well-articulated 
level within AAU's systems approach is the institutional system. As noted in the 
introduction, AAU developed a Framework for Systematic Change in Undergraduate 
STEM Teaching and Learning to guide institutional commitment to facilitate change in 
undergraduate STEM education and Essential Questions and Data Sources to measure 
progress of their reform efforts. This required identifying key levels of change, the 
agents of change, the mechanisms of change, and models for scaling and sustaining 
change. 
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Strategies for Scaling Change 


Chapter 1 Assess Organizational Strengths and 
Weaknesses 


One of the most important lessons derived from the AAU Initiative study is the importance of 
organizations engaged in change processes to start by assessing their strengths or assets to help 
devise the best and most strategic approach. Particularly for complex phenomena, there will be 
multiple organizations engaged in addressing the issue. Organizations can be most effective when 
they target their strategies to capitalize on their unique strengths. Strengths or assets can often be 
identified by exploring an organization's identity—the unique characteristics or features that define 
an organization. 


For AAU, strengths included their ability to: 
1. influence leaders such as presidents and provosts and other prestige and influence 
organizations within the overall sector, 
2. create and leverage networks, 
. define overarching logics or value systems for the enterprise, and, 
4. work across various stakeholders of the higher education system, ranging from the 
National Science Foundation to partners in STEM reform such as the Bay View Alliance to 
the entire set of AAU institutions. 


Ww 
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But an understanding of strengths will only partially serve organizations. Organizations also need 
to understand their limitations so that they might best focus their energies where they can have the 
most impact, rather than expending effort in areas unlikely to yield meaningful change. For example, 
AAU’s work to institutionalize change at the project site level was difficult given their lack of influence 
with faculty and staff, their inability to work closely with project sites to help develop their teams 
and leadership to overcome barriers, and other work that required more day-to-day relationships 
and work. 


We have created a tool for organizations to assess or map their unique strengths and assets using 
the theories of change and associated strategies described in the previous section. Particular 
strategies may be well-developed, emerging, or non-existent at your organization. You will likely 
NOT have examples of every strategy at your organization, and it is also unlikely that everything you 
do have is well-developed. That is normal. Understanding what strategies are already well-developed 
at your organization and deciding upon which new areas to prioritize is more important. 


The version below (Table 1) is partially filled out with some examples from the AAU Initiative to help 
you get an idea of how to think through this process. As an illustration, we outline how AAU used 
institutional theory. For institutional theory, influence is the major strategy used to create change. 
In order to leverage influence, we provide areas to assess—groups you might have influence with, 
type of influence you might have (e.g. incentives vs. regulation), strategies for influencing (e.g. 
communications, grants/resources), and areas of greatest legitimacy for influence (e.g. disciplinary 
standards, assessment of learning, norms for research universities). 


Groups may also choose to use other strategies or approaches not listed here. Whatever approach 
you use, it is important to consider your assets and limitations with these approaches. 


Instructions: Using Table 1, consider each theory of change and identify which strategies 


are well-developed, emerging, or nonexistent for your organization. A blank version of this 
table is available as Appendix A. 


Table 1: Asset Assessment Tool (with examples from AAU Initiative) 


Strategy is 
Nonexistent 


Strategies for 
Scaling Change 


Theories of Change Strategy is 


Well-developed 


Strategy is 
Emerging 


Have influence with 
administrators but 
not faculty 


Type of groups 


Have some 
incentives through 
grants 


Type of influence 


Institutional 
Theory/Influence 


Influence vehicles or 
strategies 


Communications, 
resource, 
competition 


Areas of greatest 
legitimacy 


Defining excellence 
in research 
university context 
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Theories of Change 


Strategies for 
Scaling Change 


Strategy is 
Nonexistent 


Strategy is 
Emerging 


Strategy is 
Well-developed 


Networking 


Partner with a 


Number of existing : 
networks foundation, a 
nonprofit and 12 
other institutions 
Type of functions Use existing 
performed by networks to raise 
existing networks funding, promote 
and market the 
change, brainstorm 
implementation, 
share information, 
and provide social 
support 
; : Have a steering 
Leadership capacity commiltce 


within networks 


mentoring program, 
succession plan, 
co-chairs for 
different networks 


Ability to connect to 


At present have not 
connected to other 


Organizational 
learning 


new networks 
networks 
Data collection Working with 
and management member campuses 
capacity to build data and 


analytics systems 
around teaching 


Venues to share 
information and data 


Vehicles of 
deliberation 


Capacity to lead 
inquiry groups 


Culture 


Legitimacy to 
articulate new value 
system 


Prestigious and 
well-respected 
among members 
and aspiring 
members 


Access to key 
stakeholders to try 
out language and 
get feedback 


Ability to craft a 
powerful message and 
frame appropriately 
for audiences 


Dissemination venues 
for communicating 
values 
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Theories of Change 


Strategies for 
Scaling Change 


Strategy is 
Nonexistent 


Strategy is 
Emerging 


Strategy is 
Well-developed 


Politics 


Ability to set an 
agenda 


Ability to coalesce 
various groups to 
work together 


Skill in information 
campaigns and 
social media 


Negotiation skills 
with groups with 
differing interests 


Lobbying and 
advocacy 


Robust advocacy 
organization within 
AAU, existing 
relationships 


Systems 


Ability to shape and 
align incentives 


Realigning 
incentives is not an 
area that AAU has 
any experience in 


Ability to work 
across and align 
departments or units 


Mapping various 
parts ofa system 
and identifying 
which aspects can be 
shaped and aligned 
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2 Ensure Distributed Leadership 


The literature on scale suggests that leadership throughout the system is critical for scaling changes. 
Interviewees noted how leadership among faculty, at the department level, at the institution, among 
AAU presidents and provosts, and within the disciplines were all needed for change to occur. 
Distributed leadership draws on leadership throughout the system and does not make distinctions 
between leaders in formal positions of authority versus informal leaders in terms of their value and 
importance for creating change. Leaders at different levels have insights into particular issues related 
to the change content as well as process. Distributed leadership includes those outside positions of 
authority including faculty, professional association employees, staff at disciplinary societies, and 
those in formal positions of authority such as administrators, disciplinary association leaders, or 
policymakers. Reform efforts would benefit from an organization facilitating the development of 
leaders at these multiple levels in support of improving undergraduate STEM education. The AAU 
Initiative partially played this role through its annual meetings that had content knowledge about 
change processes and leadership development. Distributed leadership needs support from funders, 
however, in order to develop new ways of working and managing change. 


As an organization you need to consider how you are developing leadership throughout the system 
you mapped. You may not need to create new leadership opportunities, but can link groups to 
existing leadership opportunities. 


Instructions: For each area in need of leadership development (e.g. ground-level, middle 
managers, senior leaders), identify if your organization is already doing work in this area, 
could potentially do more work in this area, or could refer organization stakeholders to 
existing resources. 


Table 2: Inventory of Leadership Development 


Area of leadership Areas of leadership Areas of leadership Existing resources to refer 
development development where we development where could | leaders for development 
are already conducting be doing more work 
work 


Bottom-up with faculty 
and staff 


In the middle among 
department chairs, deans, 
and other mid-level 
administrators 


Senior administrators on 
campus 


Faculty in disciplinary 
societies 


Staff in disciplinary 
societies 


Staff at associations 


Leaders in associations 
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Some campuses create their own leadership development programs for faculty and staff. 


We also provide some links to leadership development opportunities to which you might send 
faculty, staff, and administrators for further development. 


List of Existing Leadership Development Opportunities 


Project Kaleidoscope provides leadership development opportunities for STEM faculty and 
administrators. 

Science Education for New Civic Engagements and Responsibilities (SENCER) provides 
leadership development opportunities for STEM faculty and administrators. 

American Council on Education (ACE) offers leadership development for mostly senior 
administrators, but some leadership development is also offered to middle-level administrators such 
as deans and department chairs. 

Higher Education Resource Services (HERS) provides leadership training mostly for women 
aspiring to higher levels of leadership, but often brings in and supports mid-level administrators as 
well. It is open to both faculty and administrators. HERS also offers a leadership institute focused on 
STEM. 

American Academic Leadership Institute (AALI) offers programs and other forms of assistance 
for academic leaders in various administrative positions. 

Harvard University offers programs aimed at multiple levels of leadership for both faculty and 
administrators. 


Some programs are focused on particular institutional types: 


American Association of State Colleges and Universities (AASCU) Leadership Institute is for those 
interested in leading or aspiring to lead a state college or university. 

American Association of Community Colleges (AACC) and League of Innovation offer leadership 
development for those in the community college sector. 

The Council of Independent Colleges offer leadership training for those in the liberal arts sector. 


Certain regional groups, such as the Big Ten Academic Alliance, also offer leadership development. And 
some publishers, such as Magna Publications, focus on academic leadership and offer helpful resources 
and workshops. 


Leadership development is also offered for particular professional groups in student affairs or business 
officers; the American College Personnel Association (ACPA) Leadership Educator’s Institute and National 
Association of College and University Business Officers (NACUBO) leadership programs are just two 


examples of these. 
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3 Evaluate Framing and Language for 
Change 


An area that is also often unexamined in change processes is a consideration of any new underlying 
values being instilled alongside the change, and the language, framing, and messaging around the 
new value system. An important lesson from the institutional theory literature is that the strongest 
and most sustainable ways to scale change are to alter underlying value systems—termed the 
“institutional logics.” Organizations attempting to scale change should very carefully evaluate the 
logics they are developing (whether intentionally or not) and the language used to articulate the 
new logic. Just as individual institutions must carefully craft a vision around change for institutional 
strategic planning processes, scaled change efforts must also articulate a “common agenda” and, 
even more importantly, a compelling new set of logics to undergird institutional action across the 
sector. Our study identifies how AAU went about articulating a new institutional logic, the ways in 
which it was successful, as well as some of the areas that could have benefited from more discussion 
with stakeholders. The study identifies how attention to language can be particularly important in 
scaling changes. For example, AAU carefully developed a logic that resonated with all stakeholders— 
that AAU universities need to be as excellent at teaching as they are in research. 


Differences in language among individuals or stakeholder groups can also result in barriers or 
issues that upend change processes. The analysis conducted in the study used to inform this guide 
examined group differences to help understand how different perceptions of the role of AAU 
(e.g. understandings about AAU as an organization, interpretations of the Framework document, 
definitions of scale in terms of change) played a role in hindering the change process at various 
points. These findings suggest that checking in with constituent groups from time to time on 
core assumptions and perceptions and can be helpful to identify misunderstandings or emerging 
understandings. This seems particularly important for change processes such as this that involve so 
many different stakeholder groups. 


For example, AAU developed a Framework to support what changes would improve undergraduate 
STEM education as well as provide a process to do so. Yet the Framework was not understood by 
many involved in the Initiative. Administrators were much more likely to understand the Framework 
than faculty because of the type of language used. Faculty and some administrators, including 
department chairs, often struggled to understand the Framework in terms of what cultural 
change was involved, and how the infrastructure supported departmental work. The differences in 
perspective suggest that AAU—as well as other organizations that engage faculty and mid-level staff 
and administrators—may need to consider ways to help these groups better understand institution- 
wide work. Therefore, both communication as well as learning and development issues can emerge 
as you explore group differences. 
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Instructions: Use Table 3 to write down the change issue and language being considered 


when communicating to different groups. 


Table 3: Communicating Different Change Issues to Different Stakeholder Groups 


Issue Faculty Staff Administrators Policymakers Other— 
please list 
Importance of Relate to Relate to Relate to Relate to 
student learning | disciplinary improving demonstrating demonstrating 
outcomes standards learning accountability to accountability to 
and student policymakers and public 
experience value added for 


education to the 
public 
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Utilize Multiple Theory-Based Strategies 


Remember, theories of change are explanations for how change unfolds or progresses. The way 
change occurs can vary given the stakeholders, leadership, and organization. In projects that involve 
multiple stakeholders and complex motivations and issues, using multiple theories of action to scale 
change can be extremely valuable. Leaders in AAU adopted a multi-theory approach to the change 
process that allowed them to move forward more efficaciously Systems change, learning, influence, 
institutionalization, cultural change, and networks are varied and complex approaches to change. 


It is all too common for change efforts to adopt a more simplistic approach to change, and AAU’s 
deployment of multiple strategies, informed by various theories of action, increased its chances 
of success. Had AAU only attempted learning as a change strategy, for example, the organization 
would not have had the impact it did. Embedding strategies which can be used in multifaceted ways 
as AAU did is also a very efficient way to use time and resources, finite factors that are generally in 
low supply for change leaders. 


Here is an example of how AAU took a single mechanism — annual meetings— and infused it 
with multiple change approaches for more impact. 


AAU's annual meetings brought together all the faculty and administrators involved in the Initiative 
to share ideas across campuses and to network. In addition, AAU hosted a second meeting that 
included non-project sites and invited representatives from all 62 AAU campuses to network with 
Initiative faculty and administrators. The annual meetings were intended for campuses to share 
progress they were making on their pilot projects and to brainstorm and work through shared 
issues such as challenges in addressing reward structures. The annual meetings and workshops 
were highly regarded, as people enjoyed the opportunity to get together in person, network, share 
and learn information, and consider the various threads needed to support sustained change. These 
meetings helped develop networks, facilitate learning, influence participants, and reinforce systems 
logic and create systems thinking. Below, we review how annual meetings drew on multiple change 
approaches. 


IT/Influence. Many individuals talked about the impact of having the meetings at AAU national 
offices or in Washington, D.C. and including AAU staff and national leaders in those meetings. The 
physical space and people were noted as influential. In the words of one faculty member: “It’s hard to 
exactly pinpoint, but there was something about being in Washington, D.C. and mixing with people 
| would never get a chance to interact with, and feeling like this is so different from other meetings 
| might go to.’The prestigious nature of the speakers at annual meetings was mentioned in several 
other sections of the Scaling Improvements in STEM Learning Environments report. 


Networks. The annual meetings were a key approach to supporting and connecting various networks 
from disciplinary groups of faculty, directors from Centers for Teaching and Learning, and innovators 
in STEM. Annual meetings provided networking opportunities through socials and unstructured 
time, created networking groups comprised of faculty in particular disciplines and administrators 
in specific roles, and connected champions who were passionate about these issues across different 
AAU campuses who were often isolated or made to feel as just one of a handful of people on their 
own campuses that cared about these issues. 


Organizational learning. Many interviewees expressed how the annual meetings were important 
opportunities for learning. As one faculty member described: “I think the most successful aspect were 
the meetings—they provided us a chance to reflect on our work, to learn about what other people 
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were doing, to hear about what’s happening at the national level, and recommit to the work.’ Some 
interviewees also commented about more specific learning outcomes from the meetings, such as 
the adoption of practices from other campuses, particularly the use of data analytics pioneered at 
University of California, Davis, for example. 


Systems theory and institutionalization. Additionally, the annual meetings were an opportunity 
to reinforce a systems theory of change and institutionalization, as the meeting structure utilized 
the Framework document. Sessions also focused on specific areas within the Framework such as 
addressing promotion and tenure, improving professional development, and considering learning 
outcomes. The meetings leveraged leaders in the national movement of STEM reform and national 
organizations that could impact change at different levels and areas in the system. For example, 
the Cottrell Scholars worked with disciplinary societies through expert faculty; the Center for the 
Integration of Research, Teaching and Learning worked with a network of campuses and reached 
faculty and administrative leaders as well as emerging faculty; and the National Academy of Sciences 
shaped the national dialogue among faculty leaders. 


REFLECTION 


Given the earlier asset mapping, what are the 2-3 strength areas of your organization 
and how might these strategies be leveraged together? What mechanisms might be 
used and infused with various theories of change for greater synergy? 


Scaling Change in Higher Education: A Guide for External Stakeholder Groups 15 


pter 5 Create and Assess a Systems Approach 


Every organization will be able to work at different levels of the system to promote change. 
Some associations and organizations might be very well-prepared to work with individual faculty 
on professional development or curriculum reform efforts; other groups might be really well- 
positioned to assist mid-level leaders like deans and department chairs; other associations have 
regular workshops where they work with campuses to help institutionalize changes and are well- 
positioned to help with creating campus-level change. Some organizations can work across multiple 
parts of the system—with external groups like accreditors, with consortia of institutions and with 
individual institutions. AAU was an organization that could work at multiple levels (although not 
equally effectively at each level) and utilized this capability. Organizations should establish where 
they can work best within the overall system and strategically apply their efforts there. 


As an example from the AAU Initiative, AAU conducts much work at the external level 
of the system. AAU is a policy and advocacy organization that partners with federal 
agencies and associations in order to conduct its work. AAU staff regularly interact 
with the National Academies, National Science Foundation, and foundations. This 
allowed AAU to leverage these groups as part of the Initiative. AAU also networks 
presidents and provosts and has direct regular contact with institutional leaders at 
AAU campuses. AAU does not have regular contact with academic departments or 
with individual faculty, and this makes work at these levels more challenging. Thus, 
in mapping the system, AAU is best positioned to act at the external and institutional 
levels. Certainly, organizations can work at multiple levels, but they should recognize 
additional work may be needed to have an impact. 


Engaging and Aligning the System 

Research has shown that the more levels of the systems that are impacted, the more likely changes 
are to scale and be sustained. One of the beneficial outcomes of the AAU Initiative was thinking 
about aligning different efforts within the overall landscape of improving undergraduate STEM 
education. It is important to fund and support future work that aims to align various organizational 
efforts as Coalition for Reform of Undergraduate Education (CRUSE) is attempting to do. CRUSE is 
a Coalition focused on STEM reform that involves six national organizations and meets regularly to 
share information and plan together in order to support the overall goal of undergraduate STEM 
reform. This work to align the efforts of many organizations began during the AAU Initiative and 
needs additional resources and support to demonstrate further impact. However, the initial efforts 
were perceived to be promising. 
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Figure 2 External 
(Accreditors, 


Levels of the Consortia or 


Systems) 


System 


(Austin, 2011/2014) chapman 


(Institutional 
Leaders, 
Divisions) 


Individual 


(Faculty) 


TASK 1: Locate your work on the overall system map in Figure 2, and consider various parts 
of the system that your organization can impact. 


Instructions: Use Figure 2 above to complete each prompt below. Write where you would 
locate or place your work in reference to each level of the system illustrated in the diagram. 


1. Locate your work related to individual faculty, if any. 


2. Locate your work with departments or across groups on campus, if any. 


3. Locate your work with institutions, if any. 


4. Locate your work in terms of working with and drawing on other external groups, if any. 


5. Locate the interactions or synergy of your work across the system. Do you already work 
to build connections or alignment across multiple levels? 
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6 Leverage Influence Strategies 


In addition to a systems approach, considering your organization's influence within the system is 
also important. The literature on change has very little articulation, definition, and specific examples 
of influence strategies from an institutional theory perspective that try to shape overall norms in 
a system. The study of AAU was able to provide concrete descriptions of what influence can look 
like within higher education settings, ranging from setting up institutional competition, peer and 
benchmark comparisons, branding, awards, site visits, to partnering with influential organizations 


(see the report, Scaling Improvement in STEM Learning Environments, for more details). 


Every organization has the ability to influence some set of groups or individuals, and consideration 
of the most effective influence strategies is particularly important for a strategic approach. But 
influence is generally an implicit strategy and not one that organizations conduct strategic planning 
around—even though it is an important lever for change. This report helps articulate some of the 
influence strategies that organizations might consider. Certainly, the AAU is unique in its prestige 
and can realize influence strategies that many other organizations are not able to. However, the idea 
of planning an influence strategy and mapping out individuals, groups and organizations in which a 
change agent has influence is a generalizable strategy. 


AAU, for its part, was able to influence institutional leaders by creating competition in which leaders 
did not want to be perceived as getting behind others in a particular area of work priority. Media 
and press coverage pushed institutional leaders to prioritize teaching improvement. Seeing another 
institution in the press for addressing the quality of teaching pushed leaders to make this a priority. 
And site visits with AAU staff also directly influenced leaders. 


AAU wasalso able to influence faculty by using the name of other AAU institutions. Noting that Brown 
University or University of Pennsylvania was working on improving its teaching compelled faculty 
from other institutions to pay attention when they otherwise might not have. Faculty and academic 
departments were also compelled by a national departmental award for teaching. And departments 
found partnerships with influential groups like Howard Hughes Medical Center important and 
helpful in getting deans and department chairs’ attention—especially as groups campus leaders 
cared about were attending AAU meetings. These are just a sampling of influence strategies which 
worked at various levels of the system. 


TASK 2: Map your organization’s potential influence on Figure 2 (see p. 16). 


Instructions: Use Figure 2 to map the potential influence of your organization and the levels 
where you might have the most influence in the system. Describe the ways you can leverage 
your potential influence. Note areas where you might have significant influence, as well as 
those where you might only have some influence. 


In terms of approaches to influence, consider as a starting point the techniques used by AAU— 
external rewards, recognition and awards, media and press, site visits, partnering with other influential 
organizations, creating standards, developing benchmarks, name-dropping AAU, branding as an 
AAU site, and competition between institutions—and consider other approaches you might use. 
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1. Relevant individuals to system (e.g. faculty, student affairs staff, disciplinary or institutional 
leaders) 


a. Significant influence 
b. Some influence 


c. No current influence but important to my effort 


Approaches to influence (How can your organization go about influencing these individuals or 
groups?): 


2. Relevant departments, campus groups related to system 
a. Significant influence 
b. Some influence 


c. No current influence but important to my effort 


Approaches to influence: 


3. Institutional aspects of system 
a. Significant influence 
b. Some influence 


c. No current influence but important to my effort 


Approaches to influence: 
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Chapter 7 Build and Support Networks 


Our study of the AAU Initiative identified how networks can be extremely successful in scaling 
change, as they can serve many functions ranging from information sharing, dissemination of logics 
and new values, brainstorming, and learning, to emotional support, influence, and safety in numbers 
for risk taking, as well as resource development. Our study also demonstrated the importance of 
connecting multiple networks, providing support for networks, and developing leadership to 
maintain networks. Organizations can utilize the lessons from this study to help define and implement 
networks for scaling change. Again, organizations often organically allow networks to develop and 
may not intentionally think through how networks are utilized and connected to scaling change. The 
multiple network functions identified in the AAU Initiative would not have flourished without the 
planning and attention of AAU staff. 


AAU was careful to build and support new networks, to map and identify the type of functions 
occurring with their networks, and to conduct planning to support their networks, including building 
in informal networking time, connecting people across the networks, or communicating within and 
across the network. 


In this section, we ask you to consider your existing networks and their levels of development in 
terms of supporting change. We also ask that you use this section to brainstorm new networks and 


relationships that do not yet exist but might be important to build. 


List key groups that you network with related to the initiative you are undertaking: 


Instructions: For each of the networks listed above, fill out Table 4, mapping the various 
network features—functions and structures—in order to consider ways to improve the 
networks with which you are currently working. 


Table 4: Network Features and Strengths 


Network Feature Non-existent Emerging Well-developed 


Share information and 
build knowledge 


Build resources 


Build cultural capital 


Create relationships 


Provide emotional support 


Promote learning 


Exercise influence 


Scaling Change in Higher Education: A Guide for External Stakeholder Groups 20 


Network Feature Non-existent Emerging Well-developed 


Identify meaningful 
subgroups 


Provide informal time to 
build relationships 


Provide resources and 
means to meet and 
interact 


Provide means to 
communicate 


Provide opportunities for 
deeper and less intensive 
interaction for both weak 
and strong ties 


Get key individuals and 
opinion leaders involved 


Develop network 
leadership 


Demonstrate value of the 
network for existing and 
potentially new members 


In examining the network features of existing networks, which might need further 
refinement? 


In examining your list of networks, what new networks would you like to build? How would 
this new network support your goals? 
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r 8 Create Feedback Loops 


Creating feedback loops around change processes is important and benefits groups in efforts to 
create learning and adoption of new practices. Feedback loops are mechanisms to obtain information 
that can help adjust one’s approach or strategy. Feedback loops can also benefit other processes we 
noted, such as communication around logics, influence strategy, and network building. These would 
all be key areas for feedback. 


AAU created several feedback loops. The organization conducted regular evaluations of its meetings, 
entertained informal assessment with project site leadership, and collected data and annual reports 
from campuses as a way to provide information to support learning. 


Instructions: In Table 5 below, identify in the first column feedback loops that already exist. 
If none exist, use the second column to describe feedback loops you can create to get 
feedback on that issue. 


If you do identify feedback loops in the first column, we ask you to evaluate whether the feedback 
loops are providing adequate feedback. If not, use the second column to consider additional 
feedback loops. We provide a few examples of what we mean by feedback loops. A blank version of 
the tool is available as Appendix B. 


Table 5: Feedback Areas and Feedback Loops 


Feedback areas Existing feedback loops Creating new or adding 
additional feedback loops 


Networks and relationship Faculty surveys noted how informal Survey centers for teaching and 
development networking opportunities at meeting learning about whether they need 
were useful more networking opportunities 


Influence strategies 


Communication Other national STEM reform groups 
provided input about how other 
campuses were perceiving the Initiative 


Organizational learning 


Leadership development 


Team interactions and 
development 


Language and framing of ideas 
around change 


Information sharing 


Data use and expertise 


Incentives and rewards 


Examining links between parts of 
the system to affect change 
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Conclusion and Next Steps 


Higher education stakeholders that are external to campuses have been and can be important 
catalysts for change—particularly large scale changes. The history of higher education reflects that 
external groups have helped guide many important advances, from increased access, advances in 
research and teaching, to improved student success and new roles for campuses to play and expand 
their reach in society. In order to best maximize the role that key external organizations can play, 
our research offers insights into various planning steps that can guide this work. In this final section, 
we provide a space for reflection and summary of the key areas needed to implement scaled 
change. We hope that planning groups use this to synthesize their thinking from across the various 
sections and to consider their strategy moving forward more holistically. 


While you can start anywhere to plan your scaling strategy, we suggest the following eight areas, 
with conducting an asset assessment, ensuring distributed leadership, and framing of work being 
most helpful in the beginning, followed by a multi-theory action plan, systems planning, leveraging 
of influence, and network building. Once an overall strategy develops, we recommend mapping 
feedback loops. 


Summary and Reflection 


Instructions: Use the space below to write down the strengths and weaknesses you identified 
through completing this guidebook for each planning area. 


Planning area Strengths and weaknesses Notes 
Asset Assessment Strengths: 
Weaknesses: 


Distributed Leadership Strengths: 


Weaknesses: 


Language and Communication Strengths: 


Weaknesses: 
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Planning area Strengths and weaknesses Notes 
Systems Planning and Thinking Strengths: 
Weaknesses: 
Multi-Theory Strategies Strengths: 
Weaknesses: 
Networks Strengths: 
Weaknesses: 
Feedback Loops Strengths: 
Weaknesses: 


Scaling Change in Higher Education: A Guide for External Stakeholder Groups 


24 


Appendices 


Appendix A: Asset Assessment Tool 


Theories of Change Strategies for 
Scaling Change 


Strategy is 
Nonexistent 


Strategy is 
Emerging 


Strategy is 
Well-developed 


Type of groups 


Type of influence 


Institutional 


Theory/ Influence 
Influence vehicles or 
strategies 


Areas of greatest 
legitimacy 


Number of existing 
networks 


Type of functions 
existing networks 


Networking 


Leadership capacity 
within networks 


Ability to connect to 
new networks 


Data collection 
and management 
capacity 


Venues to share 
information and data 
Organizational 


learning 
Vehicles of 
deliberation 


Capacity to lead 
inquiry groups 
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Theories of Change 


Strategies for 
Scaling Change 


Strategy is 
Nonexistent 


Strategy is 
Emerging 


Strategy is 
Well-developed 


Culture 


Legitimacy to 
articulate new value 
system 


Access to key 
stakeholders to try 
out language and 
get feedback 


Ability to craft a 
powerful message 
and frame 


Dissemination venues 
for communicating 
values 


Politics 


Ability to set an 
agenda 


Ability to coalesce 
various groups to 
work together 


Skill in information 
campaigns and 
social media 


Negotiation skills 
with groups with 
differing interests 


Lobbying and 
advocacy 


Systems 


Ability to shape and 
align incentives 


Ability to work 
across and align 
departments or units 


Mapping various 
parts ofa system 
and identifying 
which aspects you 
can shape and align 
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Appendix B: Feedback Areas and Feedback Loops 


Feedback areas 


Existing feedback loops 


Creating new or adding 
additional feedback loops 


Networks and relationship 
development 


Influence strategies 


Communication 


Organizational learning 


Leadership development 


Team interactions and 
development 


Language and framing of ideas 
around change 


Information sharing 


Data use and expertise 


Incentives and rewards 


Examining links between parts 
of the system to affect change 
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